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Amendments to the Claims: 
This Usting of claims replaces all prior versions and Ustings of claims in the application: 

Listing of Claims : 

1-24. (cancelled) 

25. (Previously presented) The method of claim 29, wherein the surface comprises a two 
dimensional surface. 

26-28. (Cancelled) 

29. (Previously Presented) A method comprising: 

obtaining a three dimensional model of a three dimensional environment, the three 
dimensional model generated from range sensor information representing a height field for the 
three dimensional environment; 

identifying in real time a region in motion with respect to a background image in real- 
time video imagery information from at least one image sensor having associated position and 
orientation information with respect to the three dimensional model, the background image 
comprising a single distribution background dynamically modeled from a time average of the 
real-time video imagery information; 

placing a surface that corresponds to the moving region in the three dimensional model, 
wherein placing the surface comprises casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the moving region in an image plane in 
the three dimensional model, and determining a position, an orientation and a size of the surface 
based on the ray, a ground plane in the three dimensional model, and the moving region; 

projecting the real-time video imagery information onto the three dimensional model, 
including the surface, based on the position and orientation information; and 

visuahzing the three dimensional model with the projected real-time video imagery; 
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wherein identifying a region in motion in real time comprises subtracting the background 
image from the real-time video imagery information, identifying a foreground object in the 
subtracted real-time video imagery information, validating the foreground object by correlation 
matching between identified objects in neighboring image frames, and outputting the vahdated 
foregroimd object. 

30. (Previously presented) The method of claim 29, fiirther comprising tracking the 
position and orientation information of the at least one image sensor in the environment with 
respect to the three dimensional model in real-time. 

31. (Previously presented) The method of claim 30, wherein obtaining a three 
dimensional model of a three dimensional environment comprises generating the three 
dimensional model of the three dimensional environment. 

32. (Cancelled) 

33. (Previously presented) The system of claim 37, wherein the surface comprises a two 

dimensional surface. 

34-36. (Cancelled) 

37. (Previously Presented) An augmented virtual environment system comprising: 
an object detection and fracking component that identifies in real time a region in motion 
with respect to a background image in real-time video imagery information from at least one 
image sensor having associated position and orientation information with respect to a three 
dimensional model of a three dimensional environment, the three dimensional model generated 
from range sensor information representing a height field for the three dimensional environment, 
the background image comprising a single distribution backgrovmd dynamically modeled from a 
time average of the real-time video imagery information, and places a surface that corresponds to 
the moving region with respect to the three dimensional model, wherein the object detection and 



Applicant : Ulrich Neumann et al. Attorney's Docket No.: 06666-156001 /#3345A 

Serial No. : 10/676,377 

Filed : September 30, 2003 

Page : 4 of 15 

tracking component places the surface by performing operations comprising casting a ray from 
an optical center, corresponding to the real-time video imagery information, to a bottom point of 
the moving region in an image plane in the three dimensional model, and determining a position, 
an orientation and a size of the siuface based on the ray, a ground plane in the three dimensional 
model, and the moving region; 

a dynamic fusion imagery projection component that projects the real-time video imagery 
information onto the three dimensional model, including the surface, based on the position and 
orientation information; and 

a visualization sub-system that visualizes the three dimensional model with the projected 
real-time video imagery; 

wherein the object detection and tracking component identifies the moving region by 
performing operations comprising subtracting the background image fi-om the real-time video 
imagery information, identifying a foreground object in the subtracted real-time video imagery 
information, vahdating the foreground object by correlation matching between identified objects 
in neighboring image frames, and outputting the validated foreground object. 

38. (Previously presented) The system of claim 37, further comprising a tracking sensor 
system that integrates visual input, global navigational satellite system receiver input, and inertial 
orientation sensor input to obtain the position and the orientation information associated with the 
at least one image sensor in real time in conjunction with the real-time video imagery. 

39. (Previously presented) The system of claim 38, further comprising a model 
construction component that generates the three dimensional model of the three dimensional 
enviroimient. 

40-49. (Cancelled) 

41. (Previously Presented) The method of claim 29, wherein identifying a foreground 
object comprises identifying the foreground object in the subtracted real-time video imagery 
information using a histogram-based threshold and a noise filter. 
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42. (Previously Presented) The method of claim 41, wherein identifying a region in 
motion in real time further comprises estimating the background image by modeling the 
background image as a temporal pixel average of five recent image frames in the real-time video 
imagery information. 

43. (Previously Presented) The system of claim 37, wherein identifying a foreground 
object comprises identifying the foreground object in the subtracted real-time video imagery 
information using a histogram-based threshold and a noise filter. 

44. (Previously Presented) The system of claim 43, wherein identifying a region in 
motion in real time further comprises estimating the background image by modeling the 
background image as a temporal pixel average of five recent image frames in the real-time video 
imagery information. 

45. (Currently Amended) A machine-readable storage device m e dium embodying 
information indicative of instructions for causing one or more machines to perform operations 
comprising: 

obtaining a three dimensional model of a three dimensional environment, the three 
dimensional model generated from range sensor information representing a height field for the 
three dimensional environment; 

identifying in real time a region in motion with respect to a background image in real- 
time video imagery information from at least one image sensor having associated position and 
orientation information with respect to the three dimensional model, the background image 
comprising a single distribution background dynamically modeled from a time average of the 
real-time video imagery information; 

placing a surface that corresponds to the moving region in the three dimensional model, 
wherein placing the surface comprises casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the moving region in an image plane in 
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the three dimensional model, and determining a position, an orientation and a size of the surface 
based on the ray, a ground plane in the three dimensional model, and the moving region; 

projecting the real-time video imagery information onto the three dimensional model, 
including the surface, based on the position and orientation information; and 

visualizing the three dimensional model with the projected real-time video imagery; 

wherein identifying a region in motion in real time comprises subtracting the background 
image from the real-time video imagery information, identifying a foreground object in the 
subtracted real-time video imagery information, validating the foreground object by correlation 
matching between identified objects in neighboring image frames, and outputting the validated 
foreground object. 

46. (Currently Amended) The machine-readable storage device m e dium of claim 45, 
wherein the surface comprises a two dimensional surface. 

47. (Currently Amended) The machine-readable storage device medium of claim 45, 
fiirther comprising tracking the position and orientation information of the at least one image 
sensor in the environment with respect to the three dimensional model in real-time. 

48. (Currently Amended) The machine-readable storage device m e dium of claim 47, 
wherein obtaining a three dimensional model of a three dimensional environment comprises 
generating the three dimensional model of the three dimensional environment. 

49. (Currently Amended) The machine-readable storage device medium of claim 45, 
wherein identifying a foreground object comprises identifying the foreground object in the 
subtracted real-time video imagery information using a histogram-based threshold and a noise 



filter. 



50. (Currently Amended) The machine-readable storage device medium of claim 45, 
wherein identifying a region in motion in real time further comprises estimating the backgroimd 
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image by modeling the background image as a temporal pixel average of five recent image 
frames in the real-time video imagery information. 



